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ACRONYMS 


CCEA: Center for Climatic anc Environmental Aaaesament 

CRD: Crop reporting district 

CV : Coefficient of variation 

JSC: NASA Lyndon B. Johnson Space Center 

KSU: Kansas State Uni\'ersity 

LEC: Lockheed Electronics Company 

NASA: National Aeronautics and Space Administration 

NCAA: National Oceanic and Atmospheric Administration 

SRS: Statistical Reporting Service 

USDA: U.S. Pepartment of Agriculture 


TEt'T ANT FVALUATION TULN F\''R KPU AlfT CCEA PHA:5E III YIELD MODEl^ 


.‘NUMMARY 


The* four tanks ec*ve*rtfd by the* yield model test plan are ns follows. 

1. Evaluation of Kansas State University (KSU) yield models for Kansas, 
North Dakota, and three U.S.S.R. oblasts by means of 10-year biVtstrap tests 
in Kansas and North Dakota by crop reporting district (CRD) using plot model 
coefficients from all available years with CRD yield adjustments calculated 
on the basis of the years previous to the test year; 10-year Jackknife tests 
performed in three U.S.S.R. oblasts. 

C. Comparatlw test and evaluation of KSli and the Center '‘or Climatic 
and Environmental Assessment (CCEA) yield models for Kansas, North Dakota, and 
three U.S.S.R. oblasts: 10-year bootstrap tests for the KSU and CCEA Kansas, 

and North Dakota ta'dels , CRD by CRD as indicated above, but with trend effects 
prespecified; 10-year Jackknife teats performed for the KSU and CCEA U.S.S.R. 
oblast nw'dels. Emphasis is on comparing performance of the two models. 

3. Evaluation of CCEA foreign yield models: The 10-year bootstrap tests 

performed on d-'dels for which sufficient yield, weather historical data exist: 
Australia, Argentina, and Canada. Ten-year Jackknife tests performed for 

other models: Brazil, India, China, aiid the U.S.S.R.^ 

U. Evaluation of the CCEA Australia 'Argentina model form development of 
the Australia/Argentina ta?del form in Kansas: 10-year bootstrap test in 

Kansas . 

Tests and evaluations for the above tasks, including reports and recom- 
mendations, will be complete by March 1, 1977. 


Testing for the India and China models na^' be delaj'ed beyond schedule due 
to lack of data and/or difficulties in developing itiodels. Testing ns described 
will be carried out upon delivery by the CCEA of these models. 



INTROEUCTION 


Several related tasks are covered under this test plan. All tasks re- 
quire the saioe kinds of computer runs by KSU and CCEA, and all tasks must be 
accomplished on a similar schedule. It makes sense to run and to analyze the 
tests at the same time. 

Task 1: Evaluation of KSU yield models for Kansas. North Dakota, and 

three U.^.S.R. oblasts . The KSU yield models for Kansas, North Dakota, and 
the three U.S.S.R. oblasts (Tselinograd, Kurgan, and Khnel-Nttsky ) will be 
delivered in final form by December 1, 1976. The KSU model form consists of 
coefficients for plot-yield models based on selected weather-related variables, 
which apply to the entire Great Plains winter- or spring-wheat regions, and of 
CRD-specific adjustments of plot yield to CRD yield. Similar models for the 
U.S.S.R. oblasts use the appropriate Great Plains plot-yield models and 
oblast-specific adjustments. These models must be carefully tested and eval- 
uated because they represent the first advanced yield models in the lACIE 
program. 

Task 2: Comparative test and evaluation of KSU and CCEA yield models for 

Kansas. No.'th Dakota, and the three U.S.S.R. oblasts . Since the KSU models 
are being considered as replacements for the CCEA yield models for the above 
areas in April 1977 (part woj' through Phase III), the performance of the two 
models must be carefully compared. There is clearly seme overlap with Task 1. 

Task 3: Evaluation of the CCEA foreign yield models . The CCEA yield 

models for the LACIE foreign areas (operational in Phase III) must be tested 
and evaluated. Where possible, their compliance with the LACIE 90/90 criterion 
(90 percent accuracy 90 percent of the time in the final LACIE production esti- 
mate at the national level) must be tested. The foreign countries Involved 
are Canada, the U.S.S.R., Australia, Argentina, Brazil, and China. The CCEA 
Great Plains models should also be reevaluated If any changes have been made 
In them since the beginning of Phase II. 

Task U; Evaluation of the CCEA Austrslla/Argentlna Model Form . The 
Phase III Australia and Argentina models represent a model form different from 
that used in the Great Plains. A parameter based on the Palmer Drought Index 
is used as an indicator of soil moisture, which replaces precipitation. Be- 
cause the new model form is to go directly into operational use for Australia 
and Argentina, its form needs to be evaluated against the precipitation-based 
form in the U.S. Great Plains. 

The need is evident for extensivt' support by KSU and CCEA in making the 
required model nms on computers to produce results used for test data. Sup- 
port requirements are detailed later in the Test Plan. 

A working group composed of members of the National Aeronautics and Space 
Administration (NASA) , National Oceanic and Atmospheric Administration (NOAA) , 
and U.S. Department of Agriculture (USDA) has collaborated on the design of 
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the tests described here. A test evaluation camnlttee chaired by ii IfOAA rep- 
resentative and consisting of representatives from the USDA and NA:3A will eval- 
uate the test results and reconiaend model implementation or alternative action 
to the Project Management. 

It should be noted that in the United States, tests will be conducted on 
a CRD basis, even thcuc^ plans call for application in Phase III on a partition 
basis. (A "partition” is on area that is relatively uniform according to var- 
ious criteria, mainly soil type; partition boundaries, however, ore not con- 
strained to political boundaries). The reasons for using a CRD basis are that 
(l) the partitions have been defined only recently for the first time and will 
undoubtedly change at least once before April 1977, and (2) yields are avail- 
able historically for CRD's, and techniques for allocating historical yields 
among partitions have yet to be defined. 


TASK OBJECT rV'ES 


Task 1: KSU Model Evaluatio n. To establish that the LACIE 90/90 crite- 

rion is met; to evaluate the performance of the KSU model, particularly the 
accuracy and responsiveness to other-than-norraal weather conditions. 

Task 2; KSU/CCEA Model Comparison . To compare performances of the two 
models in the regions of their applicability. 

Task 3; CCEA Foreign Model Evaluation . To establish that the LACIE 90/90 
criterion is met, or to establish the actual accuracy of each CCEA foreign 
model; to evaluate performance of each model, particularly the accuracy of 
prediction and response to other-than-normal weather conditions. 

Task U; Austral! a/Arigentina Model Form Evaluation . To compare perform- 
ances oftheAustrSTa7Argentina'"^del"^om'"developeTTpecifically for Kansas 
with that of the CCEA operational Kansas model. 


EVALUATION CRITERIA 


Evaluating the performance of a yield model will be based primarily on 
the following criteria: 

1. Compliance with the LACIE 90/90 criterion or nearness to it. 

2. Response to other-than-normal weather. 



3. Mean level bias, b, over the test years. 
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U. Root mean square error, ( , over the test years, 
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The specific teats and displaya of test results that were designed to 
permit evaluation using the above criteria are discussed in the next section. 


APPROACH 


Tlie basic problem in testing yield models is to find sufficient test 
data, or alternatively, to find the best test within the constraints of 
available data. The number of available years of data (n) at a location is 
rarely enough to permit 10 years to be held apart and still allow the remain- 
ing (n - 10) years to represent adquately the effects of weather on yieJd. 

This problem exists particularly in the CCEA models. 

The CCEA yield models are built by direct regression of state historical 
reported yields using variables based on state-averaged/monthly-averaged maxi- 
mum and minimum temperatu.es and state-averaged monthly precipitation totals. 
Trend terms are specified by fitting a series of straight lines to the histori- 
cal yields over the yeai*s. A characteristic of the CCEIA models, the importance 
of which will become evident later, is that these models specify the trend 
effects based on all historical years up through 1975. The 1976 prediction 
involves an extrapolation of the trend, but tests on historical years become 
intejT»lations within a prespecified trend line. In the U.S. Great Plains, 
the models are built on stat* data but are in practice applied to each CRD 
using CRD-averaged weather data. In foreign countries, the models are built 
using data from cx^p-reporting regions roughly corresponding to states. 

The KSU models are developed from plot yields and daily weather observa- 
tions at branch experiment stations throughout the Great Plains. Over 1000 
observations went into building the wlnter-\dieat plot-yield model, and a simi- 
lar number of observations will go into the spring-wheat model. As a result, 
the winter- and spring-wheat KSU plot-yield models are "imix’ersal" for the 
Great Plains, and should also apply to foreign areas without modifications. 
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To produce a CRD-based yield model, the plot^yleld model predletlone are ad- 
justed using a ratio to best match the Statistical Reporting Set*vice (SRS) 
reported yields on a CRD basis. The process is performed in a similar manner 
for oblasts, provinces, and so forth. Therefore, the KSU model consists of a 
set of "universal" coefficients of weather-related variables and one yield- 
adjustment term for each CRD, oblast, or other region to which the model 
applies. 

The testing process must approximate as nearly as possible the way the 
models will be used operationally in predicting next year's yields. Not only 
must the tests of the KSU and CCEA models be the best possible considering the 
constraints of the available data, but they must give results which are di- 
rectly comparable. This comparability is hard to achieve because of the dif- 
ferent natures of the two models. Coefficient stability in the KSU model 
appears to be high enough that a true 10-year independent test - that is, 
holding out the last 10 years and developing coefficients only on the years 
up to (n - 10) - could be carried out. This is not possible, however, with 
the CCEA model. Therefore, the best test that can be run for the CCEA model 
necessarily sets the pattern for both model tests. 

Another factor that makes comparability particularly difficult is the 
different ways in which the trend effects are handled. The KSU model attempts 
to model the trend effects from fertiliser usage, from variety mix, and from 
fallow practices, whereas the CCEA model specifies trend empirlceQly from the 
analysis of all available years. In our tests under consideration, the latter 
practice has had the effect of prespecifying trend effects and, therefore, of 
removing them from consideration. Root mean square errors for predictions are 
those caused by weather only. Unless one postulates that he "knows" the trend 
next year is going to follow the trend over the past several years , the root 
mean square error contribution resulting f~om the uncertainty Involved in pro- 
jecting the trend from one year to next ye r is missed. To Insist that CCEA 
recalculate the trend annually, as they did in Phase-I models, is not fair 
either because this practice Introduces an additional important degree of free- 
dom not found in the operational model. The root mean square error resulting 
from such a procedure would not represent a reliable estimate of the error in 
predicting the yield for the following year. 

It appears that the fed rest way of comparing the KSU and CCEA models 
would be to effectively prespecify the trend effects as much as possible in 
the KSU model as well. This can be done by using the fertilizer-usage model 
over all the available years (or the real fertilizer amounts) and inserting 
the real-variety mix and the fallow practices for eac’ test year. Notice, how- 
ever, that this set of teat results should not be used in Task 1 (the evalua- 
tion of the KSU model). The proper test would Involve estimation of fertilizer 
usage, var .ety mix, and fallow practices for a test year only from past years. 

The best approach in testing the models to satisfy the four tasks appears 
to be the one described below. The tests simulate as closely sis possible the 
situation involved in predicting each year's yield from models developed on 
all available data to aate. 
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In situations where about 20 or more years of historical yleld/veather 
data are available, the ‘‘bootstrap” approach is used. In a 10-year bootstrap 
test, the model coefficients are developed only from the years up threu^ch 
(n - 11) and are used to predict the yield ^ r year (n - 10). Then the nn^del 
coefficients are rederived from the years ti.;cu^ (n - 10) and are used to 
predict the yield for year (n - 9) and so forth. In this manner, 10 years of 
relatively Independent tests results are produced to serve oa test data. 

In situotions where less than about 20 years of yleld/weother data are 
available for a region, a somewhat dl.’ferent approach, the "Jackknife” ap- 
proach, must be used. The (n - 10) years are insufficient to develop a reli- 
able set of model coefficients, 'therefore, one year is held apart; the model 
coefficients are developed for the remaining (n - 1) years, and tests are con- 
ducted on the held-out year. The held-out yes.- is then pemuied among the n 
available years, or among a specified subset. In particular, for a 10-year 
Jackknife test, one permutes the held-out year rjaong the last 10 years, but In 
every case develops the coefficients from the (n - l) years. Th' main draw- 
back to the Jackknife teat, as opposed to the bootstrap test, is that the trend 
terras, if not already prespeclfied as in the CCEA model, are found by inter- 
polation rather than extrapolation. This process any moke the teat somewhat 
less rigorous in predicting the performance of a nxjdel. 

The available years of data for each model to be tested, and the testing 
approach dictated, are listed below. 


Model 

Years 

Test 

U.S. Great Plains 

1920's-1975 

10 year bootstrap 

Canada 

1930's-197l* 

10 year bootstrap 

U.S.S.R. 

1958-1972 

10 year Jackknife 

Australia 

19J*0-1<J72 

10 year bootstrap 

Argentina 

1931-1970 

10 year bootstrap 

Brazil 

1962-1973 

10 year Jackkni/e 

India 

1956-1965 

10 year Jackknife® 

China 

7 

n 

10 year Jackknife 


®The exact data available and model readiness dates 
for the India and China models are not yet clear. The 
models will be tested when delivered. Tests will cor- 
respond to those described os nearly os possible. 
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Mod*ls will be tested at the same regional level at which they are used 
in LACIE. This means CRD (with aggregation to state level} in the United 
States, province in Canada, oblast in the U.S.S.R., and appropriate crop 
reporting regions In the other countries. 

The K5U will run test results for the KSU models. They are beet equipped 
to do so on a quick turnaround and will be essentially free to do so beginning 
in December. The KSU will use daily historical weather dat.. from the Kansas 
and North Dakota branch experiment stations as their test data base. Although 
the weather data is not the same as in the CCEA first-order net station data 
base, the station density (one per CRD) is similar. The KSU data base Is con- 
sidered to be entirely adequate because a point model is being compared on the 
basis of daily weather input against a state model based on monthly state 
averages. The 10-year bootstrap tests in each of the Kansas and Noi*th Dakota 
CRD*s will be run with a plot-yield model based on all available plot-weather/ 
yield data. The plot-weather/yield data constitutes a data set as nearly in- 
dependent of the met station weather and CRD yields as is probable in practice. 
The daily historical oblast yield data for the three U.S.S.R. oblasts will be 
supplied to KSU by CCEA. The CCEA will run the required tests for the CCEA 
models. These models consist of CRD^'^y-CRD x*uns for Kansas and North Dakota 
using 10-year bootstrap tests, region- -by-region runs for each LACIE foreign 
county using 10-year bootstrap or 10-year Jackknife tests, development in 
Kansas of a model of the Australian/Argentina form, and znuining a 10-year 
bootstrap test, CRD-by-CRD and statewide, to compare this model to the opera- 
tional CCEA Kansas model. 

Using the test resvilts supplied by KSU and CCEA, Lockheed Electronics 
Company (LEC) statisticians at the NASA Lyndon B. Johnson Space Center (JSC) 
will compile the following statistical parameters and displays for each CRD 
and state in the United States and for each appropriate crop reporting region 
in foreign countries. They will consider the foll owing data; The bias b over 
the test years; the root mean square error, (^ISE)^,; the plot of predicted 

yields for the two models and SRS yields versus years ; and the plot of pre- 
dicted yield versus the SRS yield over the years for each model. 

The last two considerations indicate qualitatively how well a model re- 
sponds to other-than-normal weather, euid the deviation of the least squares 
linear fit in item U from the ideal 1:1 line indicates the degree to which the 
model over- or under-responds. Response to other-than-ncrmal weather, or 
unusual relationships not monitored well by a model, can be studied by looking 
at the worst year for each model, the second worst year, and so forth in an 
attempt to identify the type of non-normal crop developments, weather pat- 
terns, diseases, and other abnormalities that the model falls to monitor. 


DATA REQUIRIMENTS 


Data required to run the tests are as follows. 
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KSU Model* 


The KSl' Bodel* (to be run at KSU) will include the following: SRS final 

yield figure* by CRD for Kan*a* and North Dakota for 19^6 (or earlier) through 

1^79 (on hand at KSU); daily hiatorical weather (T , T . , precipitation) at 

niax tain 

branch experi:aent atation* in Kanaa* and North Dakota for aane year* (on hand 
at KSU); official U.S.S.R. yield* for Taelincgrad, Kurgan, and Khnel-Nitaky 

oblaata for 1956 through 1972^ (to be aupplied to KSU by CCEA); and dally 

hiatorical weather (T . T . , precipitation) from flrat order net atationa 

ni&x nm 

in and near the teat oblaata for aane years (to be aupplied to KSU by CCEA). 


CCEA Models 

The CCEA nodela (to be run at CCEA) will Include yield and monthly his- 
torical weather data from which CCEA models were de\'eloped for each state and 

each foreign crop reporting region^ and monthly hiatorical weather data for 
each CRD in Kansas and North Dakota (on hand at CCEA). 


Statiatical Tests 

Data required to run the statistical tests (results of the bootstrap or 
Jackknife runs of yield models) are as follows. 

1. Predicted yearly yield, model parameter values, and independent data 
values for 10-year bootstrap runs of KSU models for each CRD in Kansas and 
North Dakota. These data must be run with trend effects prespecified (that 
is, using the fertiliser-usage model from all years and the real -variety mix 
and fallow practices for each test year) for use in Task 2 (to be supplied 
by KSU). 


2. Same os item 1 except that the run is made with the fex*tilizer model 
developed only from years previous to each test year and that use is made of 
projections about variety nix and fallow practices for each test year for use 
in Task 1 (to be supplied by KSU). 

3. Predicted yearly yield, rodel parameter values, and independent data 
values for 10-year Jackknife runs of KSU models for three U.S.S.R. oblasts 
(to be supplied by KSU). 

I*. Predicted yearly yields, model parameter values, and independent data 
values for 10-year bootstrap runs of CCEA models for each CRD in Kansas and 
North Dakota and for states (to be supplied by CCEA). 
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And later years if available. 
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Predicted yvarly yields, nodel paraaeter values, and Independent data 
values froB lO-year bootstrap or Jackknife runs, as appropriate, for crop re- 
porting regions of each LACII foreign country (to be supplied by CCEA). 

6. Model developed on Kansas using the Australian /Argent ini an nodel fotr, 
and predicted yearly yields, lacdel paraneter values, and independent data 
values from 10-year bootstrap runs in each Kansas CRD and for the entire state 
(to be supplied by CCEA). 

7. Predicted yearly yields, model paraaeters values, and independent 
data values fron 10-year bootstrap runs in each CRT and state for any United 
States models that have changed sii^i ficant ly from those evaluated for Phase 
II operations (to be supplied by CCEA). 


RESOURCES REQUIRED 


Estimates of manpower (Civil Service and contractor) required to accom- 
plish the testing and evaluation Jobs outlined here are as described below. 
Additional funding is not required. 

Running models required to provide test data at KSU are estimated tc re- 
quire 2 man mor* ,i I MM); at CCEA, 12 MM. 

The CO 1 .W of reducing and analyzing test reoults at JSC is estimated to 
re'j’iire S MM using LEC personnel. 

The costs involved in evaluating test results is divided between the 
costs for working group members required for the task, which is estimated at 
3 MM, and LEC support, which is estimated at 2 MM. 

Documentation of results, which would be a function of LEC support, is 
estimated at 2 MM. 


SCHEDULE 


The 

schedule 


tasks defined in this plan will be performed in accordance with the 
presented in Table 1. 
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APPENDIX 


TEST or COMPLIANCE WITH LAC IE 90/90 CRITH^ION 


Compliance teics for the CCEA modela have, to date, been conducted aa 
follovB. To determine whether a state yield model meets the LACIE 90/90 cri- 
terion at the national level in production, one considers the probability 
statement 


Pr 



^ 0.0707 

- /r 

S i 


0.90 


where P is the state production (spring or winter wheat as appropriate); 

A 8 A A 

P^ ■ A X Y and A is the SRS acreage for the state; Y is the model estimated 

yield; 6 is the fraction of national production contributed by the state, 

8 

oblast, or other location in question; and the factor 0.0707 arises from the 
product of 0.10, which corresponds to a 10-percent error (90 percent accuracy) 
times arising from the assixnption that the error is equally distributed 
betweep acreage and yield. The assvmiptions Implicit in the above are (l) 

A 

P. is an unbieised estimator of P and is normally distributed; (2) d ■ P / 

X A A 8 8 8 

P ■ var(P )/var(P), where P is the total national production; and (3) 

A 8 A 

var(P ) ■ 2 var(P, ) . 
s 1 

To devise a test for determining whether the probability statement above 
is satisfied, the following procedure is proposed. 


! 1 for X i 0 
0 for X > 0 


and define the random variable. 


z 



Pi _ P..0I0J.JP 

/r 
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Then test the null hypothesis. 


E(n»(Z') i 0.90. 


If z(t), t ■ 1, 2, ... denote independent obaervnt ions of 2, then the test ciui 
be bitsed on the followintt binominlly distributed statistic. 


Tlien let 


1 

n 


n 


t«l 


n 

s 


a Pr 



. pi /p < 0i£12l\ 

/ 


Tests of the 90/90 criterion at the state level reduce the form of the tests to 


" > 0.90 versus H : n < 0.90. 

Os— as 

A A 

The statistic for this test is p . Using the fact that n p has a binomial 
distribution, a one-tall test of H_ at the 0.07 level for n ■ 10 is to accept 

ti 1 ^ ^ 


10 > 8i 

Otherwise, reject. 

Now that estimates of LACIE acreage estimate errors are available, they 
should be used in eviluating compliance with the 90/90 criterion. To do so, 
the coefficient of variation (CV) for production is computed for each state 
using the yield errors resulting from the 10 years of tests and the acreage 
errors estimated in the latest available yeoi- (presiuncbly 197'’-76). In the 
case of the U.S.S.R. oblasts, it will be necessary to compare 1975 and 1976 
acreage estimates to the latest available reports, pn.'bably 107^. Tlie CV for 
a state la computed by ilividing the square root as the varitmce in production 
for the state by the actual (true) production in the state. The variance in 


11 



production should br calculated frore both state yield entiranteo and. If avail- 
able, from iTlD yield and acrea«te estimates a«tKrertat>id aocordlitK to LAi'IK itulde- 
llnea up to the slate level. The resultln*t CV's m\wt be scaled to the na- 
tional level imd compared to the project icca’, of t> i>ercent, ihlch cor reMpi'nda 
to the dO/dO criterion. 



